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ABSTRACT 

The usefulness of alusni ratings in assessing 
departaental quality was investigated at the University of Illinois 
by a survey one ..year after graduation, in 1977. Groups of 4,573 
enrolled students and one year later, 1,228 alusni fros 22 
departsento responded to 11 satisfaction its is concerning 
instructional, curricular, advising, and functional aspects of thes- 
es jor Atpartaents. The purpose was to assess: (1) whether the factor 
structure of alusni ratings of aajor departssnts is sisilar to that 
of enrolled student ratings; (2) vhsthsr alusni report degrees of 
satisfaction different fros thoss of enrolled students; and (3) what 
ths influences are of job -related variables on alusni attitudes 
toward their aajor progress and ths university in general. The 
results suggest that forasr students continue to asssss their aajor 
piograsi in the ease way after graduation. It was found, however, 
that aluani were sors satisfied with integration of courses, 
c la ssrooi evaluation procedure!, and accenibility of instructor than 
were enrolled students, but less satisfied with- vocational guidance. 
The alusni survsy is thus seen as a valuable source of additional 
indorsation, and is worth the additional expense to adsinister. 
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Alumni Ratings 

Abstract 

The usefulness of alumni ratings in assessing departmental quality was 
investigated. Groups of alumni and enrolled students from 22 university 
departments responded to 11 satisfaction items concerning instructional, 
curricular. advising, and operational aspects of their major departments. 
Factor structures obtained for the two groups were virtually Identical 
and included two factors: General Satisfaction with Major and Satisfaction 
with Men tor ship. Comparisons of department means showed differences between 
alumni and enrolled students on several items, particularly vocational guid- 
ance. Alumni ratings were found to be relatively uninfluenced by job- 
related variables. It was concluded that alumni ratings represent an' imp or- . 
tant source of information to be considered in assessing departmental quality. 



Alumni Ratings 

^ 1 

Alumni Ratings as an Indicator of Departmental Quality 
Alumni ratings have been used as a source of evaluative Information 
In Institutions of higher education In several different ways. First* 
alumni ratings of teaching performance for Individual professors have been 
compared with ratings of currently enrolled students. Retrospective ratings 
of alumni have beea related to current student ratings, with correlations 
ranging from .40 to .75 (Drucker & Remmers, 1951; Centra, 1974; Overall, 
Marsh, Hughes, & Unterbrlnk, Note 1). Although these ratings have a 
relatively high degree of stability, they assess only one Important dimension 
of departmental quality (i.e., individual teaching). 

A second use of alumni ratings represents a much broader approach. 
Alumni have provided their assessments of the skills needed for success In 
their current profession. Alumni from eleven Independent liberal arts 
colleges In the midwest reported. that cognitive end effective skills such 
as sensitivity % teem membership, supervision of work, and oral communication 
were Important for success but that they had not adequately learned these 
skills while In college (Rossman, Note 2). In another survey, (Ochsner 
& Solmon, 1979), most alt ?nl rated their college as useful In Increasing 
general knowledge but o nly a third rated college as very useful In Increasing 
their leadership ability and helping them choose life goals. Based on 
these surveys, Inferences about Institutional quality should take Into 
account discrepancies be twee- ?lumnl current needs and retrospective 
ratings of what they learned whU<;< T n college. 

A third, more Intermediate* us& Involves alumni assessment of the major 
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department. In a study of 25 graduate departments in history, chemistry, 
and psychology (Clark, Hartnett & Baird, Note 3), alumni were asked to 
provide information about their current professional status, career 
accomplishments, and their opinions of their college training for use in 
'program improvement. Alumni accomplishments were not found to be related 
in any clear way to measures of faculty productivity, student ability, 

« 

quality of learning Environment, or department reputation. However, 
alumni ratings of the ability of the faculty and overall excellence of the 
prograrr correlated highly C70 to .80) with ratings of enrolled students. 

Of the three uses of elumnl ratings, the greatest potential utility 
seems to lie with program end department reviews. Clark, Hartnett and 
Baird (Note 3) suggested that recent alumni have a better perspective about 
the contents, procedures, and requirements of a program than do enrolled 
students and are more objective than faculty members. In a survey of depart- 
ment heads at 134 institutions, Clark (Note 4) found that nearly 60 percent 
considered alumni ratings and opinions to be "very important" information in 
departmental reviews and evaluations for departmental use. However, only 
40 percent indicated that evaluative information from alumni was available. 

At the University of Illinois, departments are reviewed on a regular 

has La by a central faculty and adm inistrative council (Smock & — Hake, 

Note 5). A 2A-it<2a instrument titled the Program Evaluation Survey (PES) 
was developed to measure enrolled student perceptions of and satisfaction 
with Instructional, curricular, advising, and operational aspects of their 

t 

uajor department. The PES was intended to provide data both for central 
administrative staff to compare and judge departments and for departmental 
* acuity to assess strengths and weaknesses of their departments. 

\ 
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The University of Illinois also nails an alumni survey, to all degree 
recipients one year after graduation. This Instrument Includes 11 depart-' 
mental satisfaction Items taken from the PES, as well as a number of Items 
concerned with current employment and attitudes toward major program of study 
and other aspects of the University of Illinois. However, the data from 
this survey are not used as part of the departmental review process. 

Before endorsing the use of alumni ratings of departments, several Issues 
need to be addressed. First, do alumni rate along the same dimensions as they 
did 48 enrolled students? If they do, then comparisons among the ratings for 
the two groups can be made In a straightforward fashion. However, If they do 
not, caution should be used In comparing the two sets of evaluations. Second, 

0 i 

if they do rate along the same dimensions, then are alumni data worth collecting 
at all? Alumni data are certainly more costly to collect, and If the data are 
too redundant with student ratings, then they are not cost-effective. A 
third Issue concerns the Influence of one's job situation on the ratings of 
departmental quality. This Influence represents a pote-llal threat to the 
validity of the ratings and If It Is appreciable, then again the ratings must* 
be Interpreted with caution. ' 

The purpose of this paper Is to present the results of an Investigation 
of alumni ratings of departmental quality. Specifically, three major ques- 
tlons were studied^ : 

(1) Is the factor structure of alumni ratings of major departments 
similar to that of enrolled student ratings? 

(2) Do alumni report degrees of satisfaction with aspects of their 

* 

major departments which are different from these of enrolled students? 

(3) What are the Influences of job-related variables on alumni attitudes 

v 

tot jrd their major program and their university In general? 
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Method 



Instruments . 

The 1975 Program Evaluation Survey Included 1.1 items to which students 
indicated their satisfaction with various aspects of their major department. 
The satisfaction Items used a 5-polnt bipolar scale with anchor labels of 
1 - "high" and 5 - "low." 



The University of Illinois Survey of 1976 Degree Recipients Included 
the 11 departmental satisfaction Items from the PES. It also Included a 
number of Items concerned with current employment as well ss 2 Items 
inquiring about attitudes toward major program of study and the University 
of Illinois in general. These last 2 Items used a A-point bipolar scale 
with anchor labels 1 - "stron3ly negative" and A - "strongly positive." 



A total of A ,573 enrolled students (sophomore and above) majoring In 

'i 

1 of 22 academic departments from 6 colleges reviewed la 1975-76 completed 
. the PES In November, 1975. During the spring of 1977, one year after 
graduation, I+22B Alumni from the same 22 departments completed and returned 

the alumni survey. Sample sizes and return rates by department for each 

survey are shown in Tsble 1. 

Results 



Factor Structures 

Simple common factor analyses with oblique rotations were performed 



Insert Table 1 about here 



Subjects. 




on the 11 departmental satisfaction Items for enrolled students and alumni 



er|c» 



7 



Aluani Ratings 

9 

' 5 

separately. Table 2 shews that the factor structures for the two groups 
were virtually Identical. Two factors .were retained for each group. They 
were labeled General Satisfaction with Major, composed of seven it« 



(loadings > .30), and Satisfaction with Mentorshlp, composed of four 
Items. The correlation* between the factors for the .aluani and enrolled 
student groups were .65 and .63, respectively. 



Insert Table 2 about here 



Comparisons Between Rating Groups 

Twenty-two^depertment means were computed separately for the alumni 
and enrolled student groups on each of the 11 satisfaction items. Two . 
departments were deleted due to unstable alumni means caused by small 
(n < 10) sample sices. Table 3 shows the comparisons between the department 
means for each item. Highly* significant correlations (p < .01). were found 

♦ 

for six Items, five of which belonged to the General Satisfaction with Major 
factor. In general, the correlations were higher for Items from this factor 
than for those from the Satisfaction with Mentorshlp factor, 



> from 



Insert Table 3 about here 



Dependent t- tests were performed on the 20 department means and are 
also shown in Table 3. Four Items showed significant (p < .01) differences. 
For three -of the Items (Integration of courses, classroom ev uiation proce- 
dures, and accessibility of instructors) the alumni group reported greater 
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satisfaction, while for ths fourth Item (vocational guidance) the alumni 

► 

were much less satisfied (p < .0001). 
Influences of Job on Alumni Attitudes 

Multiple regression analyses were performed on the Individual aluanl 
data. The two dependent variables were items on the alumni survey concerning 
current overall attitudes toward one's majorprogram and toward the Univer- 
slty of Illinois. Two sets of predictor variables were used; the 11 PES 
items and 8 job-related variables. A total of 490 "unemployed" alumni ware 
deleted from the regression analyses, since they did not complete the items 
related to jobs.. The majority of these alumni classified as unemployed 
were enrolled In either graduate or professional school. 

Table 4 shows the product-moment correlations between overall attitudes 
and the two variable sets. The correlations with overall attitude toward major 
program were consistently higher than those with overall attitude toward 
the university. AIsql, the correlations wltL. the PBS satisfaction items were 
generally higher than with the job-related variables. 



Insert Tables 4 and 5 about here 



The multiple regressions of the variable seta on the overall attitudes 
are_ahownJti_Table 5. The results are consistent for both overall attitudes. 
The job-related variables showed high multiple correlations^- du e p rlmarlly____ 
to the item dealing with helpfulness of major in job. However , th« -multiple 
correlations for the PES items were substantially higher than with the job- 
related variables and the Increments in the multiple correlations when the 

9 
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job-related variable* were entered after the PES items were negligible. 

Discussion 

TWo factors of departmental satisfaction were found for both alumni 

and enrolled students: General Satisfaction with Major and Satisfaction 

with Men tor ship. These factor structures were highly consistent with 

.'those found by Lrajtkaap, Vise and Hengstler (1979), who used a revised version 

of the PES which contained a similar set of departaental satisfaction Items. 

The results of Braskamp et al. (1979) and this study suggest that former 
» . ' * 

students continue to evaluate their major program along the same dimensions 

* r 

after graduation. Hence It appears reasonable to mske comparisons between 
departmental ratings of enrolled students and alumni. 

This raises a cost-effectiveness concern. Unless substantial dif- 
ferences are found between enrolled student and alumni ratings, .the col- 
lection of the more expensive alumni ratings may not be necessary in order 

* • 

to evaluate departmental quality. For the majority of the PES satisfaction 
items, high correlations were found between department means for the two 
rating groups. This Is consistent with the findings of Clark et al. (Rote 

' Hoveuer, significant differences were found between department means 
for four Items. Alumni were more satisfied with integration of courses, 
classroom evaluation procedures, and accessibility of Instructors than' 
were enrolled students, whereas they were less satisfied with vocational 
guidance . —While the meaning of the first three differences will -ba left — 
open to Interpretation, the meaning of the vocational guidance difference, 
•etas clear. Alumni have a nore valid perspective on the quality of voce— 

» 

1G 
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tional guidance In a department than do enrolled students. Only when 
students graduate and enter the working world can the f y be t*p£cted to^ 
reasonably evaluate the vocational guidance, .they have received. While 
enrolled students represent' a unique perspective on assessing departmental 

quality (Braskamp et 41.; 1979), alumni ratings are sufficiently nonredun- 

• 

dant with enrolled student ratings to suggest that alusnl also offer a 
unique perspective in evaluating* certain aspects of departments. 

A possible undesirable influence on aJLumni ratings concerns the 
employment situation of individual alunni. If these ratings of a depart- 

m 

I 

ment are to, be valid they should not be unduly influenced by factor* such 
aa job satisfaction or salary which are not directly relevant to the 
evaluation of departmental quality. > 

The results of the multiple regression analyses showed that while 
the job-related variables correlate highly aa a set with alumni attitudes 
toward both major program and' the university, the dominant predictor 
was the variable dealing with helpfulness of major in job* Of 'the job- . 
related variables, this is the most relevant to departmental quality. 
A common goal of departments is to provide training for students to per- 
fo.rm competently In their Jobs, Hence, the substantial correlations 
between alumni attitudes and helpfulness of major In job should not be 

surprising. The other' job-related variables did not show very high v 

« 

correlations with alumni attitudes. 

On the other' hand, the PES satisf action items showed higher correla- 
tions with alunni attitude toward major program, and slightly lower 

\ 

4 11 



Alumni Ratings ' 
9 

'correlations with attitude toward the university. Moreover, the contri- 

» 

bution to prediction of the set of job-related variables beyond that of 
the PES satisfaction items was virtually zero. Thus, the job-related 
variables accounted for very little of the variance of alumni attitudes 
that- was not accounted for by the PES satisfaction Items. 

In evaluating departmental quality, it Is desirable to Include as 
many relevant sources of data as possible. Just as enrolled student 
ratings represent a relevant perspective on departmental quality, this 
study has demonstrated that alumni ratings can also provide a unique source 
of data in assessing'.departments. 

■ \ 

% 

* • 

i 



12 



Aluani lutings 
10 

Reference Notes 

1. Overall, J.U., Marsh, H.W., Hughes, R. , and Uitterbrlnk, H.L. Lona- 

term stability of students' cvaluatlona: A longitudinal study . 
Paper presented at -annual meeting of the Association for Institu- 
tional Research, Houston, May, 1978. 

2. Rossaann, J.E. The emoh a ais on affective and cognitive outcomes in 

an alyses of student development. Paper presented at annual meeting, 
of Association for Institutional Research, Eouaton, May l'>78. 

3. Clark, M.J., Hartnett, R.T., & Balrd, L.L. Assessing Dimensions of 

Quality in Doctoral Education . Pi^nceton, N.J.: Educational Testing 
Service, 1976. 

"4. Clark, M.J. Program Review Practices of University Departmanta . 

Princeton, N.J. : ORE Board Research Report CREB No. 75, July, 1977. 
5. Smock, H.R. I Ilaka, H.W. • COPE: A systematic approach to .the evaluation 
of academic departments .' Paper presented at annual meeting of Ameri- 
can Educational Research Association, New York City, April, 1977. 



} 



13 



ER?C, 



• m 



Alumni Ratings 
11 



References 



Braekamp, L.A. , Wise, S.L. and Hengstler, D.D. Student satis fact ion as 

a measure of departmental quality. Journal of Educational Psychology . 

1979, 71, 494.-498. 
Centra, J.C. The relationship between student end alumni ratings of 

teachers. Educational and Psychological Measurement . 1974, 34, 321-325. 
Drucker, A.J. and Remmers, H.H. Do alumni and students differ in their 

attitudes toward instructors? Journal of Educational Psychology . 195®, 

42, 129-143. 

Ochsner, N.L. and L.C. Soltaon. College Education and Employment; The 
Recent Graduates . Bethlehem, PA: The CPC Foundation, 1979. 



14 



Alunml lutings 

Table 1 

12 

Sample Sizes and Survey Return Rates for 
PES and Alumni Surveys by Major Department 



Instrument 
PES , Alumni Survey " 



Major Department Sample Size , X Return Sample Size X Return 



Agronomy 


110 


68 


27 


68 


Animal Science 


257 


72 


55 


67 


pCod^Stience 


121 


73 


19 


68 


Health and Safety Education 


124 


68 


43 


73 


Leisure Studies 


228 


81 


47 


55 


Physical Education 


450 


82 


77 


68 


Economics 


350 


79 


68 


56 


Educational Psychology 


151 


66 . 


75 


. 68 


Elementary Education 


503 


64 


,214 


73 


Architecture 


603 


54 


169 


60 


Dance 


102 


99 


6 


50 


Landscape Architecture 


126 


72 


31 


65 


Music 


537 


62 


145 


57 


Theatre 


105 


36 


21 . 


62 


ui Don rx tinning 


1 19 
X JZ 


OX 


11 

JX 


an 

oy 


Labor & Industrial Relations 


44 


67 


11 

i 


61 


Ecology, Ethology, & Evolution 


65 


75 


4 


100 


Linguistics 


73 


91 


10 


63 


Microbiology 


109 


72 


47 


70 


Physiology & Biophysics 


119 


72 


29 


91 


Speech Communications 


* 

105 


58 


34 


61 


Speech and Hearing Science 


159 


79 


65 


72 


Combined Departments 


4573 


69 


1228 


65 
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Table 2 

Factor Loadings of 11 PES Satisfaction 
Items for Alumni and Enrolled Students 



Program Evaluation Survey Item 


Program Evaluation Survey (PES) Factors 


General Satisfaction 
with Major 


Satisfaction with 
Mentorship 


Alumni 


Enrolled 
Students 


■• 

Alumni 


Enrolled 
Students 


Challenge of program 


.79 


.70 






Integration of courses 


.67 


.62 






Quality of Instruction 


. o / 




m 




Texts and Instructional materials 


.65 


.51 


V 




Classroom evaluation procedures 


.48 


.40 


• 




Accessibility of Instructors 






.55 


.49 


Academic advising 






.80 


.74 

• 


Vocational guidance 






.66 


.76 


Faculty-student communication 






.80 


.71 


Worth of program 


.70 


.76 






Overall satisfaction with program 


.76 


.78 







Note: Only loadings greater than ;30 are reported. 
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Tabic 3 

Comparisons Between Department Means of Enrolled 
Students and Alumni for 11 PES Satisfaction Items 





Average Department Mean 


Average Department 






Program Evaluation Survey (PES) Item 


for Enrolled Students 


Mean for Alumni 


Correlation 




Challenge of program 


2.27 


2.24 


.65** 


0.23 


Integration of courses 


* 

2.67 


2.49 


.79** 


2.88** 

e* • w 


Quality of instruction 


2.42 

w 


2.34 


.78** 


1.76 


Texts and instructional materials 


4 2.53 


• 2.44 


.55** 


1.68 


Classroom' evaluation procedures 


2.78 


2.50 


.37 


4.47** 


Accessibility of instructors 


? 20 


1.77 


. zz 




Academic advising . 


2.58 


2.74 


.49* 


-1.80 


Vocational guidance 


7.91 


* 

3.27 


.39 


-5.50** 


Faculty-student communication 


2.62 


2.61 


.58** 


0.07 


Worth of program 


• 1.96 


2.07 


.38 


-1.97 


Overall satisfaction with program 

* 


2.22 

r 

•3» 


2.23 


.81** 


-0.18 



* - p < .05 

** - p < .01 , 

Mote; The ratings range from 1 (high satisfaction) to 5 (low satisfaction). A total of 20 departments were used. 



Table 4 



Correlations of Overall Alumni Attitudes 



with Job-Related Variables and PES Items 



Alumni Ratings 
15 



Overall Attitudes 



Variable Sets 



Toward Toward the 

major program University of Illinois 



Job-related Variables 
Employment Status 
Relation between job and major 
Helpfulness of major In job 
Job satisfaction 

Underemployed In terms of salary 
Underemployed In terms of riisponai* j r 
Underemployed In terms of Jc u a& ^ urt\c\ 
Salary 
PES Savfr faction Items 
ChaiUm*;.* of program 
Integration of courses 
Quality of Instruction 
Testr and it :;: <: ructlonal materials 
Classroom evaluation procedures 
Accessibility of Instructors / 
Academic advising 
Vocational guidance 
Faculty-student communication 
Worth of program 

Overall satisfaction with program 



-.01 
-.28 
.42 

!1 



. X'> 

, !.?-. 
. A." 



.38 
,30 
.44 
.40 
.36 
.59 
.68 



-.01 
-.13 
.23 
.13 
.09 
.11 
.09 
.10 

-.33 
-.40 
-.36 
-.33 
-.31 
-.29 
-.37 
-.28 
-.28 
-.42 
-.49 



ERIC 



Note: A total of 490 unemployed alumni were deleted from these analyses, 
leaving a sample size cf 738. 
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Multiple Regressions of Overall Alumni 

■ 

Attitudes Using Job-related Variables and PES Items 



Variable Sets 



Overall Attitudes 



Toward 



major program 
„2 



Toward the 



University ot Illinois 



R 



R 



R 



I. Job-related variables . .435 .189 

II. PES satisfaction items .700 .490 

III. Increment of set I after set II * .021 .000' 



.251 
.527 
.008 



.063 
.278 
.000 



